Skip to content

Updated the python dependencies (targeted for Beam 2.63.0)#33658

Merged
liferoad merged 4 commits intoapache:masterfrom
liferoad:beam-update-python
Jan 21, 2025
Merged

Updated the python dependencies (targeted for Beam 2.63.0)#33658
liferoad merged 4 commits intoapache:masterfrom
liferoad:beam-update-python

Conversation

@liferoad
Copy link
Contributor

@liferoad liferoad commented Jan 18, 2025 •
edited
Loading

Fixes #33639.

Following https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-HowtoupdatedependenciesthatareinstalledinPythoncontainerimages:

./gradlew :sdks:python:container:generatePythonRequirementsAll

and https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters, updated the base image deps.

Some notable package updates:

  • numpy==2.0.2 and pandas==2.2.3 for Python 3.9 (note numpy >=2.1 does not support py39)
  • numpy==2.2.1 and pandas==2.2.3 for Python 3.10, 3.11, and 3.12
  • protobuf==5.29.3 for all Python versions

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@liferoad liferoad marked this pull request as ready for review January 18, 2025 15:39
@liferoad liferoad changed the title Updated the python dependencies Updated the python dependencies (targeted for Beam 2.63.0) Jan 18, 2025
@liferoad
Copy link
Contributor Author

liferoad commented Jan 18, 2025 •
edited
Loading

For :sdks:python:test-suites:tox:py312:testPython312:


apache_beam/dataframe/pandas_doctests_test.py:1073: AssertionError |  
  | ----------------------------- Captured stdout call ----------------------------- |  
  | ********************************************************************** |  
  | File "array", line 94, in array |  
  | Failed example: |  
  | pd.array(['a', 'b'], dtype=str) |  
  | Expected: |  
  | <NumpyExtensionArray> |  
  | ['a', 'b'] |  
  | Length: 2, dtype: str32 |  
  | Got: |  
  | <NumpyExtensionArray> |  
  | [np.str_('a'), np.str_('b')] |  
  | Length: 2, dtype: str32 |  
  | ********************************************************************** |  
  | File "array", line 103, in array |  
  | Failed example: |  
  | pd.array(['a', 'b'], dtype=np.dtype("<U1")) |  
  | Expected: |  
  | <NumpyExtensionArray> |  
  | ['a', 'b'] |  
  | Length: 2, dtype: str32 |  
  | Got: |  
  | <NumpyExtensionArray> |  
  | [np.str_('a'), np.str_('b')] |  
  | Length: 2, dtype: str32 |  
  | ********************************************************************** |  
  | File "array", line 182, in array |  
  | Failed example: |  
  | pd.array([1 + 1j, 3 + 2j]) |  
  | Expected: |  
  | <NumpyExtensionArray> |  
  | [(1+1j), (3+2j)] |  
  | Length: 2, dtype: complex128 |  
  | Got: |  
  | <NumpyExtensionArray> |  
  | [np.complex128(1+1j), np.complex128(3+2j)] |  
  | Length: 2, dtype: complex128 |  
  | ********************************************************************** |  
  | File "array", line 193, in array |  
  | Failed example: |  
  | pd.array([1, 2], dtype=np.dtype("int32")) |  
  | Expected: |  
  | <NumpyExtensionArray> |  
  | [1, 2] |  
  | Length: 2, dtype: int32 |  
  | Got: |  
  | <NumpyExtensionArray> |  
  | [np.int32(1), np.int32(2)] |  
  | Length: 2, dtype: int32 |  
  | ********************************************************************** |  
  | 1 items had failures: |  
  | 4 of  16 in array |  
  | ***Test Failed*** 4 failures. |  
 

<br class="Apple-interchange-newline">

@github-actions
Copy link
Contributor

github-actions bot commented Jan 18, 2025

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @shunping for label python.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@tvalentyn
Copy link
Contributor

tvalentyn commented Jan 21, 2025

For :sdks:python:test-suites:tox:py312:testPython312:

I see that all tests are passing - is it still an issue?

I might be risky to make this change in 2.63.0 given that we won't have much time to test the changes before the release train starts, hence, higher risk of upgrade issues in 2.63.0; i would target this for 2.64.0

@liferoad
Copy link
Contributor Author

liferoad commented Jan 21, 2025

For :sdks:python:test-suites:tox:py312:testPython312:

I see that all tests are passing - is it still an issue?

I might be risky to make this change in 2.63.0 given that we won't have much time to test the changes before the release train starts, hence, higher risk of upgrade issues in 2.63.0; i would target this for 2.64.0

:sdks:python:test-suites:tox:py312:testPython312: flaky for some dataframe tests. Fine for now

I think it is not risky since we do not directly depend on pandas and numpy and the critical probuf version is already upgraded to 5.x with 813ef7c 3 weeks ago.

@tvalentyn
Copy link
Contributor

tvalentyn commented Jan 21, 2025

LGTM

@liferoad liferoad mentioned this pull request Jan 21, 2025
Closed
17 tasks
@liferoad
Copy link
Contributor Author

liferoad commented Jan 21, 2025

#33697 is opened to track the flaky doctest issue.

jrmccluskey
jrmccluskey approved these changes Jan 21, 2025
View reviewed changes
Copy link
Contributor

@jrmccluskey jrmccluskey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, as discussed offline hopefully we'll find a better way to keep these in sync with dependabot PRs moving forward

@liferoad liferoad merged commit 9afc92e into apache:master Jan 21, 2025
96 checks passed
Abacn added a commit that referenced this pull request Jan 27, 2025
@Abacn
Revert "Updated the python dependencies (targeted for Beam 2.63.0) (#…
23d72e8 
…33658)"

This reverts commit 9afc92e.
@Abacn Abacn mentioned this pull request Jan 27, 2025
Closed
@Abacn
Copy link
Contributor

Abacn commented Jan 27, 2025

numpy 2 still fails a test, see #33774 (comment)

This was referenced Jan 27, 2025
Closed
Closed
@shunping shunping mentioned this pull request Feb 11, 2025
Merged
VardhanThigle pushed a commit to VardhanThigle/beam that referenced this pull request Mar 21, 2025
@liferoad @VardhanThigle
Updated the python dependencies (targeted for Beam 2.63.0) (apache#33658
aae6142 
)

* Updated the python dependencies

* skip pd.array checks

* minor updates

* remove the skip and it seems flaky
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jrmccluskey jrmccluskey jrmccluskey approved these changes

@tvalentyn tvalentyn Awaiting requested review from tvalentyn

@damccorm damccorm Awaiting requested review from damccorm

@kennknowles kennknowles Awaiting requested review from kennknowles

Assignees

No one assigned

Labels

docker Next Action: Reviewers python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: Update Beam containers to numpy 2.x

4 participants

@liferoad @tvalentyn @Abacn @jrmccluskey